pyxis: allow unprivileged user namespaces so enroot works on Ubuntu 23.10+ - #1392

Agreed on all three points — clearing kernel.apparmor_restrict_unprivileged_userns host-wide removes exactly the attack surface the Ubuntu default was added to reduce, and the original patch did it unconditionally.

What changed (commit 86ecc83):

• The role now installs an AppArmor profile that grants the userns capability to /usr/bin/enroot-nsenter alone and leaves the host default in place.
• The host-wide sysctl remains available but is now an explicit, documented, default-off choice: pyxis_userns_allow_globally, false by default. If the profile fails to load, the play stops with the parser error and points at that variable rather than silently falling back to the weaker setting.
• Both paths stay a no-op where /proc/sys/kernel/apparmor_restrict_unprivileged_userns does not exist, and the profile path additionally requires apparmor_parser.

Why the profile only covers enroot-nsenter:

$ grep -l CLONE_NEWUSER /usr/bin/enroot-*
/usr/bin/enroot-nsenter

Checked against enroot 3.2.0-1, so enroot-mount, enroot-switchroot, enroot-mksquashovlfs and enroot-aufs2ovlfs do not need it.

Evidence so far, on a DGX B300 running DGX OS 7.5 (Ubuntu 24.04.4, AppArmor 4.0 ABI available):

$ apparmor_parser -Q -T -v enroot-nsenter
Addition succeeded for "enroot-nsenter".

What is still missing, and I would rather say so than leave it implied: I do not have a container launch on a host where the restriction is active. This node was already worked around out-of-band before I picked up the issue — /etc/sysctl.d/91-enroot.conf, not owned by any package, sets the knob to 0 — so it is not in the affected state. Restoring the restriction to reproduce the failure would break the Pyxis jobs currently running on it (an MLPerf Inference run occupying all 8 GPUs).

I will add the before/after launch output once the node is free. Please hold the merge until then if that evidence is required — I would rather this sit than be merged on a parser check alone.

The node is free now, so here is the container launch evidence I said I would add.

To produce it I had to put this host into the affected state first: it carries an out-of-band /etc/sysctl.d/91-enroot.conf that sets the knob to 0, so Pyxis works here for the wrong reason. I moved that file aside, set the knob back to 1, ran the launch, installed the profile, ran the same launch again, and restored the original state afterwards.

Environment

OS:              Ubuntu 24.04.4 LTS
kernel:          6.8.0-106-generic
enroot:          3.2.0-1
apparmor_parser: AppArmor parser version 4.0.1
hardware:        DGX B300, 8x SXM

Before — restriction active, no profile

$ sysctl kernel.apparmor_restrict_unprivileged_userns
kernel.apparmor_restrict_unprivileged_userns = 1

$ srun --partition=test --gres=gpu:1 --cpus-per-gpu=8 --time=5 \
      --container-image=ubuntu:24.04 -- sh -c 'echo container-started-ok'
pyxis: importing docker image ...
error: pyxis: container start failed with error code: 1
error: pyxis: printing contents of log file ...
error: pyxis:     enroot-nsenter: failed to create user namespace: Permission denied
error: pyxis: couldn't start container
error: spank: required plugin spank_pyxis.so: task_init() failed with rc=-1
error: Failed to invoke spank plugin stack
srun: error: dgx01: task 0: Exited with exit code 1

exit status: 1

Install the profile

$ apparmor_parser -r /etc/apparmor.d/enroot-nsenter
$ echo $?
0

$ aa-status | grep enroot
   enroot-nsenter

After — same launch, profile loaded, restriction still active

$ sysctl kernel.apparmor_restrict_unprivileged_userns
kernel.apparmor_restrict_unprivileged_userns = 1

$ srun --partition=test --gres=gpu:1 --cpus-per-gpu=8 --time=5 \
      --container-image=ubuntu:24.04 -- sh -c 'echo container-started-ok'
srun: job 136 queued and waiting for resources
srun: job 136 has been allocated resources
pyxis: importing docker image ...
container-started-ok

exit status: 0

The knob reads 1 during both launches. That is the part I wanted on the record: the container starts with the OS security default left in place, which is what your review asked for and what the original patch did not do.

Afterwards the profile was unloaded and removed, 91-enroot.conf was put back, and the knob returned to its original value of 0, so the node is as it was.

Two notes on the run itself. The image is plain ubuntu:24.04 rather than one of our local squashfs images, so the reproduction does not depend on anything specific to this site. And pyxis_userns_allow_globally stayed at its default of false throughout — the global sysctl path was never taken.

All three convergence gaps are addressed in eec281e.

1. Reconcile the effective profile on reruns

The load was gated on enroot_userns_profile.changed alone. The kernel's own list now decides as well:

- name: check whether the enroot-nsenter profile is loaded
  command: grep -q "^enroot-nsenter " /sys/kernel/security/apparmor/profiles
  register: enroot_userns_loaded
  changed_when: false
  failed_when: false
  when: ...

- name: load the enroot-nsenter AppArmor profile
  command: apparmor_parser -r /etc/apparmor.d/enroot-nsenter
  register: enroot_userns_load
  failed_when: false
  when:
    - ...
    - (enroot_userns_profile.changed | default(false))
      or (enroot_userns_loaded.rc | default(1) != 0)

Evidence

Three consecutive runs, with the kernel profile list stubbed to model each state:

Same three runs against this branch as it stands:

2. apparmor_parser absent is no longer a silent skip

- name: install the AppArmor userspace (Ubuntu)
  apt:
    name: apparmor
    state: present
  when:
    - is_compute
    - ansible_distribution == "Ubuntu"
    - apparmor_userns_knob.stat.exists | default(false)
    - not pyxis_userns_allow_globally

- name: fail when the AppArmor userspace is unavailable
  fail:
    msg: >-
      ... the scoped enroot-nsenter profile cannot be loaded and enroot would
      fail with "failed to create user namespace: Permission denied". Install
      the AppArmor userspace, or set pyxis_userns_allow_globally=true to clear
      the restriction for the whole host instead.
  when:
    - ...
    - not apparmor_parser_bin.stat.exists | default(false)

The profile is the default path, so its userspace is a dependency of the role rather than something to skip over. If it is still missing after the install, the play stops instead of reporting success while enroot keeps failing.

3. Turning the fallback off takes its setting with it

- name: drop the host-wide user namespace sysctl when the fallback is off
  ansible.posix.sysctl:
    name: kernel.apparmor_restrict_unprivileged_userns
    sysctl_file: /etc/sysctl.d/60-enroot-userns.conf
    state: absent
  register: enroot_userns_sysctl_removed
  when:
    - is_compute
    - apparmor_userns_knob.stat.exists | default(false)
    - not pyxis_userns_allow_globally

- name: reapply the remaining sysctl configuration
  command: sysctl --system
  changed_when: true
  when: enroot_userns_sysctl_removed.changed | default(false)

Removing the line alone would leave the running kernel at the lowered value until the next boot, so the remaining configuration is reapplied. sysctl --system was chosen over writing a value back because it restores whatever the distribution ships rather than a number this role invents.